\section{Future Work}

Due to the time limitation in this project, we feel there is still much room to improve in the current system. Here we list several major ones.
\begin{itemize}
\item Though our implementation has considered the possibility of changing the number of map and reduce tasks according to the load of the system; this feature is not fully supported. As shown in the last two experiments, varying the ratio between numbers of map tasks and reduce tasks can affect the processing performance of the system. This means that it is possible to develop a measure to quantitate the system load and use that as a feedback to JobDispatcher to decide an optimal configuration for running map tasks and reduce tasks in accordance to the current input pressure. 
\item Our current work only supports one Mapreduce job at a time in the system. While a typical streaming application usually consists of multi-stage of operators. It is also unknown if it is straightforward to transfer stages of streaming operations into native Mapreduce jobs. Even so, we need another layer built on top of Hadoop to specify and guarantee the data dependencies between stages in streaming.
\item One of the reasons that the Mapreduce provides fault tolerance is its usage of distributed file systems. We discarded it in support of the streaming process. Consequently our current design does not have any fault tolerance abilities. If one task fails, there is no way to recover the data since they are not preserved in disks. Adding checkpointing or keeping data until it is fully safe to discard would be the options to provide fault-tolerance in our current system. Actually, HDFS is so closely-coupled with Hadoop that we feel we could as well implement streaming in Mapreduce from scratches.

\end{itemize}

Another direction to combine Mapreduce and streaming is to implement Mapreduce in a streaming architecture such as IBM's System S system. Similar work has been done for several other architectures\cite{he08} \cite{Mrcell}. It would be worthwhile to study the performance of Mapreduce in a streaming processor.
